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This is in response to the appeal brief filed 4/23/2004. 
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(1) Real Party in Interest 

A statement identifying the real party in interest is contained in the brief. 

(2) Related Appeals and Interferences 

A statement identifying the related appeals and interferences which will directly affect or 
be directly affected by or have a bearing on the decision in the pending appeal is contained in the 
brief. 

(3) Status of Claims 

The statement of the status of the claims contained in the brief is incorrect. A correct 
statement of the status of the claims is as follows: 
This appeal involves claims 1-5 and 7-20. 
Claims 6 and 21-31 have been canceled. 

(4) Status of Amendments After Final 

The appellant's statement of the status of amendments after final rejection contained in 
the brief is correct. The amendment after final rejection, as filed on 4/23/2004, and which 
cancels claims 21-31, has been entered. 

(5) Summary of Invention 

The summary of invention contained in the brief is correct. 
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(6) Issues 

The appellant's statement of the issues in the brief is substantially correct. The issues, as 
adjusted, are as follows: 

(a) Whether Claims 1-5 and 7-12 are unpatentable under 35 U.S. C. §103 as being 
obvious over Oberteuffer et al. (U.S. Patent No. 6,438,523) in view of Kono (U.S. Patent No. 
5,914,707). 

(b) Whether claims 13-20 are unpatentable under 35 U.S.C. §103 as being obvious 
over Oberteuffer et al. in view of Kono. 

(7) Grouping of Claims 

Appellant's brief includes a statement that claims 1-5 and 7-20 do not stand or fall 
together and provides reasons as set forth in 37 CFR 1. 192(c)(7) and (c)(8). 

(8) Claims Appealed 

The copy of the appealed claims contained in the Appendix to the brief is correct. 

(9) Prior Art of Record 

6,438,523 Oberteuffer et al. 8-2002 

5,914,707 Kono 6-1999 
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(10) Grounds of Rejection 

The following ground(s) of rejection are applicable to the appealed claims: 
Claims 1-5 and 7-20 are rejected under 35 U.S.C. 103(a). This rejection is set forth in a 
prior Office Action, mailed on 1/28/2004. 

(11) Response to Argument 

Regarding claim 1, the appellant argues that the specific correspondence between plural 

input modes and plural output modes, as expressed in claim 1, cannot be anticipated by the 

combined teachings of Oberteuffer et al. (hereafter referred to as "Oberteuffer") and Kono, since 

neither reference teaches implementing multiple input modes and multiple output modes. The 

appellant thus appears to suggest that, for claim 1 to be anticipated, a single reference must teach 

implementing multiple input modes and multiple output modes, as the appellant asks, 

Since Oberteuffer et al. does not envision multiple output modes, and 
Kono does not envision multiple input modes, how can plural input-output 
correspondences be suggested by either? (See page 3, lines 14-15 of the 
appeal brief). 

In response to such a piecemeal analysis of the references, the Examiner notes that claim 1 was 
not rejected by either Oberteuffer or Kono, but instead was rejected by the combined teachings 
of Oberteuffer and Kono. While it is true that neither Oberteuffer nor Kono explicitly presents a 
device with multiple input and multiple output modes, the Examiner nevertheless maintains that 
Oberteuffer and Kono, when combined, teach outputting content graphically in response to 
graphical input, and outputting content audibly in response to audio input. Without question, 
Oberteuffer demonstrates outputting content graphically in response to graphical input (for 
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example, see column 4, lines 37-53). Thus the main issue regarding the patentability of claim 1 
is whether or not Oberteuffer and Kono teach outputting audio content in response to audio 
input, and whether Oberteuffer and Kono may be combined to teach such a feature. The 
Examiner believes so, as Oberteuffer explicitly teaches audio input, and Kono teaches outputting 
audio content in response to such input. For example, Oberteuffer demonstrates inputting 
commands via speech (see column 4, lines 54-61). Kono teaches responding to commands by 
outputting content comprising audio (for example, see column 3, line 55 - column 4, line 35; and 
column 7, lines 28-32). Thus contrary to the appellant's assertion that u [t]he proposed 
combination [of Oberteuffer and Kono] would merely result in allowing a user to input data 
using voice or handwriting... and then have the data played backed simultaneously aurally and on 
screen. . ." (see page 3, line 19 - page 4, line 2 of the appeal brief), the Examiner maintains that 
the proposed combination further teaches responding to graphical user input with content 
comprising graphical content, and responding to audio user input with content comprising audio 
content. 

Further concerning claim 1, the appellant asserts that, although Kono describes a pocket 
organizer which can be used as a dictaphone, Kono does not teach a particular correspondence 
between particular input modes and respective output modes. The Examiner agrees, for if Kono 
did teach a particular correspondence between particular input modes and output modes, the 
Kono reference by itself would be enough to anticipate claim 1. Rather, the submission that 
Kono discloses a device which may receive speech input provides further evidence of the 
similarity between the Kono device and the Oberteuffer device, which receives speech input in 
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the form of text and commands (for example, see column 4, lines 27-62 of Oberteuffer). Such a 

similarity serves to support any motivation to combine Oberteuffer and Kono. 

As per motivation, the appellant asserts that the proffered suggestion to combine 

Oberteuffer and Kono, that "audio output further enhances understanding of graphically 

displayed content," bears no relevance to Oberteuffer, which displays no book content at all 

The appellant particularly submits, 

. , . [A]ll Oberteuffer et al. does is accept voice input and handwriting input 
for digitizing them into computer files, something that nothing in the prior 
art, particularly Oberteuffer et al, suggests must be improved by 
providing multi-mode output. In other words, the user of Oberteuffer et 
al does not seek to access content from anything, but rather to input 
content already known to the user into a computer file, thus requiring no 
multi-mode repetitive output of what the user has just input. (See page 4, 
lines 14-18 of appellant's appeal brief). 

In response the Examiner notes that, even assuming the device of Oberteuffer is unable to 

display book content (an idea with which the Examiner does not necessarily agree), the device of 

Oberteuffer still necessarily displays data, such as text data. For example, Oberteuffer discloses 

that the user may create documents of various types by entering text and commands via speech 

and handwriting (for example, see column 4, lines 17-62). It is well understood that the device 

of Oberteuffer has the additional capability of storing and displaying such documents, for the 

documents would be of no use if they could not be accessed and viewed. Along the lines of such 

functionality, Kono describes a similar type of device: 

... [A] compact microcomputer notebook or pocket organizer is useful to 
collect dictated communications, such as, notes relative to an overseas trip 
and the user may visually read written words or sentences that appear on 
the organizer's display, but for such a small slim device there is no facility 
to orally reproduce the voice pronunciation of the words and sentences 
due to its compactness. To include concurrent audio reproduction 
capability and user inquirable and inquisitorial interfacing while retaining 
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their compact size is considered not practical under present technology for 
such organizers. (See column 1, lines 42-52). 

Kono teaches improving these devices by providing audio output in synchronization with other 

data, and interfacing to allow random access to this audio and other data: 

...[T]he employment of an optical medium relative to this invention 
renders it possible to display a large amount of data in the form of 
alphanumerics, simple diagram and animation graphics and to reproduce 
audio data, such as sound, voice and music, being synchronized in 
presentation with the displayed data. The present invention provides a 
simple, compact and portable electronic apparatus for recording visual and 
audio data, randomly retrieving such data from the optical memory and 
synchronizing the displayed data with the audio reproduced data. (See 
column 4, lines 21-32). 

Thus, contrary to the appellant's assertion that nothing in the prior art suggests that Oberteuffer 
must be improved by providing multi-mode output, Kono describes a similar type of device, and 
suggests it may be improved by providing multi-mode output in the form of synchronized 
graphical and audio data. Such teachings are certainly relevant to the device of Oberteuffer, 
which like the prior art device described by Kono, has the functionality of collecting dictated 
communications and visually displaying written words on a display. The proffered suggestion to 
combine Oberteuffer and Kono is consequently considered to bear relevance to Oberteuffer. 

Further regarding the Examiner's rejection for claim 1, particularly regarding whether the 
device of Oberteuffer is able to display content other than what the user enters into it, the 
appellant alleges that although the device comprises a web browser, such a web browser is only 
used to create, edit and view documents. The appellant thus concludes that one would not have 
been motivated to combine Oberteuffer with Kono, which concerns displaying data other than 
what the user enters into it. Even assuming the device of Oberteuffer is unable to display content 
other than what the user enters into it (an idea with which the Examiner does not necessarily 
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agree), the teachings of Kono are still applicable to Oberteuffer, since as described in the 
previous paragraph, Kono teaches providing a multi-mode output with a device that explicitly 
displays content that the user puts into it. The Examiner maintains, additionally, that the web 
browser of Oberteuffer is able to download content, particularly content not created by the user. 
Such functionality is well-known in the art, and certainly more well-known than the appellant's 
alleged use of a web browser to create and edit documents. 

In reference to claim 13, the appellant argues that neither Kono nor Oberteuffer suggest 
maintaining a current position in content and/or a spanning region of the content being rendered 
such that first and second output threads are run simultaneously with each other, and whereby as 
recited in claim 13, an abstract interface determines positions at which tangible interfaces should 
pause when required to coordinate the multiple output modes. The appellant particularly attacks 
the Examiner's interpretation of the Kono reference, stating that the Examiner's allegation that 
Kono teaches pausing audio or video output is incorrect. As evidence, the appellant submits that 
Kono discloses that audio output can footnote or exemplify displayed text, citing col. 7, lines 30- 
33 of Kono, and concludes that "[t]his does no implicate the need to pause anything because the 
content of the audio footnote, while related to the text, is not the same as the text and, hence, 
only a loose coupling such as starting both displays simultaneously need be implemented." (See 
page 6, line 15 - page 7, line 3 of the appellant's appeal brief). While it is true that Kono 
describes audio output which can exemplify displayed text, the Examiner maintains that either 
the display of the text, or the output of the audio, is paused in order to synchronize the text and 
audio. For example, Kono discloses that when an audio passage serves as a footnote, it is within 
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the context of an electronic book comprising a multitude of pages (see column 6, line 66 - 
column 7, line 39). An audio passage is associated with a particular page, and played when an 
icon corresponding to the audio passage is selected (see column 7, lines 28-32). By having the 
audio and text of an electronic book divided into such pages, the audio and text are 
synchronized; the display of an electronic book is paused on a single page, and the audio passage 
directly associated with that page is played. Likewise, if an audio passage associated with a 
particular page is being played, and the user advances to a new page, it is understood that the 
audio passage would be paused, as it is no longer associated with the new page. The positions at 
which the audio and graphical interfaces should pause in order to coordinate the multiple output 
modes are thus determined, each position specifically corresponding to a page of the electronic 
book. 

Further regarding claim 13, the appellant argues that there is no teaching or suggestion by 

Kono or Oberteuffer that user input commands can be received from an audio user interface, 

whereby a graphical user interface is updated in response thereto, and vice- versa. The appellant, 

however, fails to provide any support regarding such an argument, other than stating that "[t]he 

Office Action indulges in some pretty creative bending of the references to conjure up this part 

of claim 13." The appellant fails to specifically address the Examiner's evidence, as presented in 

the Office Action of 1/28/2004, and which is repeated as follows: 

In regard to claim 13, the processor of the above-described electronic book 
of Oberteuffer and Kono is considered an "abstract interface," like that of 
the claimed invention, wherein this processor is responsible for accessing 
content stored in the data storage of the electronic book as is known in the 
art. In addition, Oberteuffer discloses that the electronic book may receive 
commands via its audio user interface, whereby it updates its graphical 
user interface in response thereto (for example, see column 5, lines 57-65). 
Kono similarly teaches that the electronic book may receive commands 
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via its graphical user interface, whereby it updates its audio user interface 
in response thereto. Specifically, it is understood that commands to 
change a graphically displayed page of a document may be entered via the 
graphical user interface, whereby the speech interface is updated in order 
to reflect the currently displayed page (see column 7, lines 29-39). It is 
therefore understood that the above-described electronic book taught by 
Oberteuffer and Kono comprises: content stored in a data storage; an 
abstract interface accessing the data storage; an audio user interface 
communicating with the abstract interface; and a graphics user interface 
communicating with the abstract interface, the abstract interface receiving 
user input commands from the audio user interface and updating the 
graphics user interface in response thereto, the abstract user interface 
receiving user input commands from the graphics user interface and 
updating the audio user interface in response thereto. (See page 8, lines 5- 
21). 

In light of such support, the Examiner maintains that the combination of Oberteuffer and Kono 
teaches receiving user input commands from an audio user interface, whereby a graphical user 
interface is updated in response thereto, and vice-versa. 



For the above reasons, it is believed that the rejections should be sustained. 
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